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(57) ABSTRACT 

A compact, handheld imaging apparatus which can be used 
to c apture nigh-quality iris image s for identification of a 
perso n. The handheld iris imager is non -invasive and non- 
contacting and comprises a camer a, a cold mirror, a lens, and 
an illuminator. The imager 'has sensors and indicators which 
assist a user in aligning and focusing the device. The imager 
also automatically captures the image when proper position- 
ing is achieved. A template of the image is then compared to 
a database of previously stored templates of images to 
i dentify the perso n. The imager is integrated into a telephone 
as a security module. The telephone cannot be unlocked and 
used unless a user has been identified and authorized by the 
imager. 

33 Claims, 6 Drawing Sheets 
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IRIS IMAGING TELEPHONE SECURITY 
MODULE AND METHOD 

HELD OF THE INVENTION 

The present invention relates in general to identification 
of physical characteristics of a human being or other animal. 
More particularly, the present invention relates to iris rec- 
ognition. 

BACKGROUND OF THE INVENTION 

Various technologies are used for uniquely identifying a 
person in accordance with an examination of particular 
attributes of either the person's interior or exterior eye. One 
of these technologies involves the visual examination of the 
particular attributes of the exterior of the iris of at least one 
of the person's eyes. The iris of the human eye has random 
patterns of striations, ciliary processes, crypts, rings, furrows 
and other features which had been shown capable of gen- 
erating highly unique bio metric templates for personal iden- 
tification. In this regard, reference is made to U.S. Pat. No. 
4,641,349, "Iris Recognition System", issued to Flom et al., 
and U.S. Pat No. 5,291,560, "Biometric Personal Identifi- 
cation System Based on Iris Analysis", issued to Daugman. 
As made clear by these patents, the visible texture of a 
person's iris can be used to distinguish one person from 
another with great accuracy. Thus, iris recognition can be 
used for such purposes as controlling access to a secure 
facility or a bank automatic teller machine, for example. An 
iris recognition system involves the use of an imager to 
video image the iris of each person attempting access, and 
image processing means for comparing this iris video image 
with a reference iris image on file in a database. 

Iris identification systems have been developed that are 
capable of collecting images of the iris and processing them 
to produce biometric templates. These templates may be 
used to identify individual irises with extremely low error 
rates, on the order of 1 in 10 78 . The systems capture the iris 
images using stationary optical platforms that are often 
large, complex, and expensive. The systems are difficult to 
use with minimal cooperation of the subject being identified. 
As a result their usefulness in many applications is limited. 

The cellular telephone industry each year loses an esti- 
mated $650 million to cellular fraud, principally due to 
cloning of cellular telephones. Cloning involves 
re-pro gramning a phone's electronic serial number and 
telephone number to those stolen from a legitimate sub- 
scriber. To counteract cloning, some service providers have 
utilized personal authentication techniques such as personal 
identification numbers (PIN) or voice verification to verify 
that the authorized subscriber is using the phone. This is 
unreliable because PIN number can be stolen or forgotten, 
and voice verification messages may be recorded. 

Although the art of human recognition systems is well 
developed, there remain some problems inherent in this 
technology, particularly the lack of an iris imager and 
security module suitable for integration into a cellular 
telephone, and the lack of a method for using biometric 
information for enabling access to the cellular network. 
Therefore, a need exists for a recognition system that 
overcomes the drawbacks of the prior art. 

SUMMARY OF THE INVENTION 

The present invention is directed to a telephone security 
module comprising: iris acquisition means having a front 
surface for obtaining an image of an iris of an eye; a lens 
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having a image plane disposed in front of the front surface 
of the iris acquisition means; a mirror disposed on a side of 
the lens opposite the iris acquisition means; an illuminator 
disposed along a side of the mirror; a memory for storing an 

5 iris image obtained by the iris acquisition means; a processor 
for extracting a template from the stored iris image; and a 
communications interface for transmitting the template to a 
central station. 
According to one aspect of the invention, the iris acqui- 

10 sition means comprises a camera, and the mirror is a cold 
mirror. The camera is sensitive to light having a wavelength 
in a range between about 400 nm and about 1100 nm. The 
mirror reflects light having a wavelength in a range between 
about 400 nm and about 700 nm and passes light having a 

15 wavelength greater than about 700 nm . 

According to another aspect of the present invention, the 
illuminator emits light having a wavelength in a range 
between about 680 nm and about 900 nm towards the iris of 
the eye being imaged, and the eye is out of contact with the 

20 iris imaging apparatus. 

According to another aspect of the present invention, the 
module further comprises at least a visible indicator or an 
audible indicator to indicate when the image of the iris has 

^ been obtained. According to another aspect of the present 
invention, the module further comprises a focus assessment 
processor coupled to the visible indicator and/or the audible 
indicator. 

According to another aspect of the present invention, the 

30 processor unlocks a telephone responsive to a signal 
received from the central station. 

In a further embodiment within the scope of the present 
invention, a method of unlocking a telephone responsive to 
the identification of a person comprises the steps of: (a) 

35 storing image information of the iris of at least one person's 
eye; (b) illuminating an eye of an unidentified person having 
an iris; (c) obtaining an image of the iris of the unidentified 
person; (d) determining if the image is an image of sufficient 
quality for a step (f) of comparing; (e) repeating steps (b) 

40 through (d) until the image of sufficient quality is obtained; 
(f) comparing a template of the obtained image with the 
stored image information to identify the unidentified person; 
and (g) unlocking the telephone responsive to a result of the 
step of comparing. The stored image information used for 

45 identification can be a code or template extracted from the 
image, and the comparison can be performed at a central 
database maintained by a telephone service provider. 

According to one aspect of the present invention, the 
method further comprises the step of activating an indicator 

50 if the image is of insufficient quality. The indicator is an 
audible indicator. 

According to another aspect of the present invention, the 
method further comprises the step of activating an indicator 
if the image is of sufficient quality. The indicator is a visible 

55 indicator. 

According to another aspect of the present invention, the 
method further comprises the step of activating an indicator 
responsive to the step of comparing. The indicator is a 

60 visible indicator. 

In accordance with a further aspect of the present 
invention, the step of determining if the image is an image 
of sufficient quality comprises the step of focus assessment 
processing the image. 

65 According to another aspect of the present invention, the 
telephone is unlocked if the step of comparing identifies the 
person. 
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In a further embodiment within the scope of the present FIG. 10 is a flow diagram of an exemplary method of 

invention, a system of identification of a person for unlock- operation of a telecommunications iris imager in accordance 

ing a telephone comprises a telephone security module, a with the present invention, 
second memory for storing at least one template of at least 

one image of an iris of at least one person's eye; and a 5 DESCRIPTION OF EXEMPLARY 

second processor for comparing the template of the stored EMBODIMENTS AND BEST MODE 

iris image with the at least one stored template of the second ^ present invention is directed to a compact, handheld 

memory to identify the person, and for unlocking the tele- imaging apparatus and method which can be used to capture, 

phone responsive to the result of the comparison. high- quality iris images. Preferably, the imager has sensors 

In accordance with a further aspect of the present to anc j indicators which assist the human operator in aligning 

invention, the second memory and second processor are and focusing the device. The imager also automatically 

housed in a central station remote from the telephone captures the image when proper positioning is achieved, 

security module. Because it is small and compact, it is practical for integration 

In accordance with a further aspect of the present into a cellular telephone where i t is^used to authenticate, 

invention, the second processor unlocks the telephone when 15 telephone su hscrjhers and _ehmmate'cellular_tfckp hone fraud, 

the template of the stored iris image substantially matches Throughout the following detailed description similar ref- 

the at least one stored template of the second memory. erence numbers refer to similar elements in the figures of the 

In a further embodiment within the scope of the present drawings, 

invention, a telephone security module comprises: iris FIG. 1 illustrates a preferred embodiment of the handheld 

acquisition means having a front surface for obtaining an 20 imager 100 in accordance with the present invention. The 

image of an iris of an eye; a lens having a image plane exemplary handheld, non -invasive, no n- contacting iris 

disposed in front of the front surface of the iris acquisition imager comprises iris acquisition means 105, an imaging 

means; a mirror disposed on a side of the lens opposite the lens 110, a mirror 120, an optional diopter correction lens 

iris acquisition means; an illuminator disposed along a side 125, and an illuminator 130. The imager 100 is preferably 

of the mirror; a first memory for storing at least one template 25 powered by a standard DC supply provided by a cellular 

of at least one image of an iris of at least one person's eye; telephone. 

a second memory for storing an iris image obtained by the The iris acquisition means 105 is preferably a conven- 

iris acquisition means; a processor for extracting a template tional solid state video camera, such as a charged coupled 

from the stored iris image; and a comparator for comparing device (CCD) or complementary metal oxide semiconductor 

the template from stored iris image with the at least one (CMOS) device. A preferred camera is a VS inch format, 

template to identify the person. monochrome CCD board camera, such as Computar Model 

EM200. Preferably, the video camera 105 is sensitive to light 
of wavelengths in the range of about 400 nanometers to 

The foregoing and other aspects of the present invention 35 about 1100 nanometers, and is positioned so that its front 

will become apparent from the following detailed descrip- surface coincides with the image plane of the lens 110 in 

tion of the invention when considered in conjunction with front of it. In the preferred embodiment, the object plane of 

the accompanying drawings. For the purpose of illustrating the lens is approximately 89 mm in front of the lens 110. 

the invention, there is shown in the drawings an embodiment More preferably, the lens 110 is an optical lens with approxi- 

that is presently preferred, it being understood, however, that ^ mately 14.2 mm focal length. 

the invention is not limited to the specific methods and The mirror 120, preferably a concave cold mirror having 

instrumentalities disclosed. In the drawings: a radius of curvature preferably about 276 mm, is disposed 

FIG. 1 is a schematic diagram of an exemplary iris imager on the side of the lens 110 opposite the video camera 105 and 

in accordance with the present invention; creates a magnified virtual image of the iris behind the 

FIG. 2Ais a schematic diagram of the imager of FIG. 1 45 muror 120. In the preferred embodiment, the mirror 120 

shown in greater detail; reflects visible light with wavelengths in the range of about 

FIG. 2B is a schematic diagram of another exemplary 400 <° about 700 nanometers, and passes light having longer 

imager in accordance with the present invention; ^inSteB " '"^ * 

FIG. 3 is a simplified flowchart of a method of operation ^ „, . _ A 

in accordance with the present invention; 50 The illuminator 130 ^ positioned just outside the edge of 

. . , - .... the cold mirror 120 and is used to illuminate the ins of the 

FIG. 4 is a schematic diagram of an exemplary ins image .. ... ,. c , ™ f .... . . ian 

..... j .... r *. subject being identified. The preferred illuminator 130 emits 

recognition system in accordance with the present invention; «. / , . & , c r , ^ on , , . nrtn 

to J . light having wavelengths or about 680 to about 900 nanom- 

FIG. S is a schematic diagram of an exemplary ins imager cters Prcf6rablVj ^ illuminator 130 is a miniature quartz 

having visual and aural indicators in accordance with the SJ halogcn or gas bu]b operating at a p proxima tcly 1 

present invention; watt 

FIG. 6 is a more detailed flow chart of a method of The { ifes { % of ^ ifis with sufficient 

operation m accordance with the present invention; clarity> focus? ^ size fof use witfa image 

FIG. 7 is a schematic diagram of an exemplary iris image processing and comparison routines, preferably in less than 
recognition system having a focus assessment processor in 60 aD0Ut 3 seconds. A preferred image processing and corn- 
accordance with the present invention; parison routine is described in U.S. Pat. No. 5,291,560, 

FIG. 8 is a schematic diagram of an exemplary iris imager "Biometric Personal Identification System Based on Iris 

incorporated into a telephone in accordance with the present Analysis", issued to Daugman, and commonly assigned with 

invention; the present invention to IriScan Inc. , and incorporated herein 

FIG. 9 is an isometric view of an exemplary telecommu- 65 by reference. However, any processing and comparison 

nications iris imager and telephone in accordance with the technique can be used with the image that is acquired at the 

present invention; and imager, such as the image pixel correlation technique 
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described in U.S. Pat. No. 5,572,596, "Automated, Non- by the mirror 120 and forms an approximately centered 

Invasive Iris Recognition System and Method", issued to image 109 of the eye 150 at the image plane 107 of the 

Wildes et aL and the techniques described in U.S. Pat. No. camera 105. The image is then captured and processed, as 

4,641,349, "Iris Recognition System", issued to Flom et aL, described below. 

both of which are incorporated herein by reference. 5 Although a cold minor (one which reflects shorter wave- 

FIG. 2A shows the apparatus of FIG. 1 in greater detail. lengths and passes longer wavelengths) is described herein, 

The lens 110 gives a high resolution image of the eye 150 of 11 15 ™derstood that a hot mirror (one which reflects longer 

the user, who is positioned in front of the lens 110, so that wawfenglhs and passes shorter wavelengths) could also be 

extreme proximity between the eye 150 and the imager 100 uscd , 1D accordance m Z™^ n D ^ ch a 

t -at . *• j jl » 4 . 6 . . , 1fl configuration is shown man imager 101 in FIG. 2B. The eye 

is not required (i.e., no contact is needed between the subject 10 ieft t ... . , . . n - # ™ vT 

a th • inn\ 15 ^lu^iDated by an illuminator 131 emitting light 

ana me imager iuu> haviag wavelengths in ^ range of about 680 to 900 

The handheld iris imager comprises a solid-state image nanometers. This light is reflected by the eye 150 and the 

capture device and an optical system which forms an image light having wavelengths in the range of about 700 to 900 

109 of the iris on the image capture device at the image plane nanometers is reflected by the hot mirror 121 to be focused 

of the video camera 105 and at the same time produces a 15 by the lens 111 onto the front surface of the camera 106. 

virtual image 115 of the iris which the user can use to Light reflected from the eye 150 having shorter (visible) 

position and focus the iris image. As a result, the user can, wavelengths in the range of about 400 to 700 nanometers 

using the same eye being imaged, see a reflected image of passes through the hot mirror 121 and strikes a concave 

the iris which can be used to position the handheld imager broadband mirror 122 which reflects light having wave- 

100 so that a good iris image (i.e., an image that can be 20 length from about 400 to 900 nanometers. This light forms 

processed and compared to those stored in a database) can a virtual image 115 of the eye 150 behind the concave mirror 

be obtained. 122 that the user can see and use to align and focus the 

FIG. 2A also shows an optional dioptric correction lens device, as described below. 
125 positioned between the eye 150 and the cold mirror 120. The imager 100 of FIGS. 1 and 2A, as well as the imager 
The dioptric correction lens 125 is an adjustable optical of FIG. 2B, is used in a system to identify the iris image that 
element which corrects for the close-range focusing ability has been captured. As shown in FIG, 3, the eye is illuminated 
of the individual eye, which varies from subject to subject. at step 160, and an acceptable or suitable image of the iris 
When the lens 125 is properly adjusted, the magnified, is obtained at step 165. The image is processed to extract an 
reflected virtual image 115 of the subject's eye appears in 3Q iris template or code at step 170, the template or code is 
sharp focus to the subject at the same eye-to-mirror distance encrypted (optional) and transmitted to the cellular provider 
at which the subject's eye is sharply focused on the front (such as a central station; e.g., a Mobile Telephone Switch- 
surface of the camera. This simplifies use of the imager, ing Office) at step 175, and the template or code is decrypted 
because the subject simply positions the image so that the (if necessary) and compared to pre-existing templates or 
virtual image 115 of the iris appears sharply focused. 35 codes of authorized subscribers stored in a memory or 

Apreferred embodiment of the dioptric correction mecha- database for identification and authorization of the user at 

nism has no correction lens 125 and instead has a mechani- step 180. If the user is authorized, the cellular provider 

cal means (not shown) for adjusting the position of the cold enables the call placement at step 185. The cellular provider 

mirror 120 relative to the camera lens 110. This allows the can either enable the call at the central station or send a 

user to vary the object distance of the cold mirror 120, thus ^ signal to the telephone processor directing it to unlock the 

changing the eye-to-lens distance at which the virtual image telephone. 

115 of the iris is sharply focused. FIG. 4 is a schematic diagram of an exemplary iris image 
The ability to set the dioptric correction mechanism to recognition system in accordance with the present invention, 
accommodate a particular user has a great utility if the The imager 100 is coupled to a microprocessor 210 that 
imager is used by only one person most of the time. Once the 45 performs the processing and encryption. The microprocessor 
correction is set, the user can easily position the device to 210 resides in a cellular telephone 200. 
obtain a sharply focused reflected image. This automatically The microprocessor 210 is coupled to the imager 100 via 
produces a sharply focused image from the camera and conventional cables and/or printed circuit boards (PCBs) 
substantially immediate acceptance of the image by the that are incorporated into the telephone 200. Other conven- 
focus assessment processor described below. Image capture 50 tional means for coupling the imager 100 and the micro - 
time is thereby reduced and overall convenience and utility processor 210 can be employed. The microprocessor 210 
is enhanced. controls the imager 100 and runs software held in read only 
An eye 150 is positioned in front of the imager 100 (e.g., memory (ROM) 205. The processor 210 is connected via a 
about 3.5 inches in front), as shown in FIG. 2A, and the bus 207 to the ROM 205, a random access memory (RAM) 
illuminator 130 is turned on. This, in turn, illuminates the 55 232, another memory such as an erasable programmable 
eye 150 and the iris therein. Preferably, the light having ROM (EPROM) 230, and an input/output (I/O) controller 
wavelengths of about 400 to about 700 nanometers is 2 25. The RAM 232 is large enough to hold at least one 
reflected by the cold mirror 120, thereby forming a magni- captured image of an iris. The I/O controller 225 is con- 
fled virtual image 115 behind the mirror 120 which the user nected to the appropriate circuitry and drivers (not shown) 
can see through the eye being imaged. The radius of 60 f° r issuing commands to control the imager 100. 
curvature of the mirror is selected so that the magnified The imager 100 preferably uses a digital camera and 
image 115 of the eye substantially fills the user's entire field transmits digital images directly to the processing unit 210. 
of view. Hence, when the imager 100 is positioned so that "On/off" data is transmitted from the imager 100 to the 
the entire eye 150 is visible, it is virtually assured that the processor 210 to initiate the image acquisition function. A 
eye 150 will be substantially centered in the object plane 140 65 digital image could be provided if a digital camera is used, 
of the camera 105. Under these conditions, the light having The image processing consists of a number of image 
wavelengths of about 700 to about 900 nanometers is passed processing steps (such as those described in U.S. Pat. No. 



08/12/2002, EAST Version: 1.03.0002 



US 6,377,699 Bl 

7 8 

5,291,560 and U.S. Pat. No. 5,572,596, which are herein If the ima ge is not suitable at.step 360. then_at_st sp.38(k 

incorporated by reference) which lead to extraction of a the Jndj£atQr(s) , is activated (e.g., a beep sound is issued), 

unique and highly specific digital biometric template that and proc essing continues at step 355 (i.e., another image is 

can be used to identify the individual based on intensity obtainlH).— ^* — ' 

patterns within the iris. The biometric template is transmit- 5 tf Because the eye's own focusing system automatically 

ted to the cellular provider where it is compared against adjusts to bring the virtual image 115 into sharp focus to the 

other templates stored in a memory or database. The data- user, it cannot be relied upon to always accurately focus the 

base stores selected data representing images of the iris of a eve image 0D camera 105. For this purpose, a focus 

plurality of subjects. A match of the biometric template with assessment system is used in one embodiment, as shown in 

a template stored in the database identifies the subject whose 10 FIG ; 7 - Digital video image information from the imaging 

iris is being imaged. device 100 is stored in a frame buffer memory 410, such as 

Although an image of the eye is reflected back to the sbB ^ t to ^ f 2 dcscribcd above ™ th r 5 s P ect to r 

subject in mirror 120, this may not provide the desired ™ . 4 aad ca P a f blc of st ™J ODe °° m P lcte franle ° f 

feedback to the user to enable the user to properly position ^ g „ lllzed v.deo information. A focus assessment processor 

the imager so that a suitable iris image is obtained. For M 42 « acccsses the digitized miagp ^formation and applies 

example, a user may be a novice in using and positioning the certam algorithms wkch are disclosed in a 

imager 100 with respect to the eye 150, or the user may be fP^ing application entitled "Video-Rate Focus 

attempting to image the eye of another subject with the ^™ o q '^m C °™ y ^ ^ ^ff 011 S ? r ' 

imager. Thus, preferably, the imager 100 comprises a pas- N °" 60.«>9^60 Nov. 25 1998 and incorporated herein by 

sive feedback mechanism to guide the user in positioning the » reference. The output of the focus assessment is used to 

eye 150 to an optimum location to allow acquisition of a ?° *? mdlca,or ' such 25 edible indicator 310. As 

suitable image g as f° cus assessment processor 420 determines that 

. ' „ , , ~the-captured jmage is not acceptable for further processing 

The passive feedback mechanism is an indicator or com- and compariso^tfeauaible^ndicator 310 is directed to emit 

binauon of indicators that provides, on a near real-time riodic sounds t0 ^ ftTSS&^faagM are repeatedly 

basis, an indication to the user that an adequate ins image 25 acquiled and taauied wtil an accep ti bk on6 is received 

has or has not been obtained. FIG. 5 is a schemata diagram ^ an acceptable iris has been reC eived, (ne audib i c 

of an exemplary ins image recognition system that includes mdicator 310 is tumed off ^ ^ Md j fa rctained fof 

posuion indicators in accordance with the present invention. pr0C e Ssin g ^ comparison, for example, by the 

Preferably the indicator is visible and/or audible, such as, microprocessor 210, as described above, 

tor example, an mdicator lamp 305 (e.g., a light emitting A i . i. • r • r * 

j ■ . /icr,vi.u .i- k. u . li • i. l Anv known technique for image focusing can be used 

diode (LED)) hat bghts when an acceptable image has been ^ ^ - of ^ * ^ ^ as 

captured (i.e., image acquired ), and a aural mdicator via a described m * g _ p ^ ^ 

speaker 310, such as a beep or other tone that sounds ^ to ^ Measurcmen RoutiDe ' s ^ ^ t ScanQcrs « 

periodically until an accep able image has been captured ^ {Q fi v $ p Nq * ^ 

(,e ' unagmgin progress ). Adjustment Device Having Restricting Means for Restrict- 

Additional indicators 306, 307 can be also be used, either ing a Selecting Action According to the Degree of Nearness 

alone or in combination, for such indications as "subject of a DistanC e Measurement", issued to Tokunaga et al., and 

identified-accept" and "subject not identified-reject". These tj. S . p a t. No. 5,404,163, entitled "In-Focus Detection 

mdications would be activated pursuant to the results of the ^ Method and Method and Apparatus Using the Same for Non 

processing and comparison performed at the database server Contact Displacement Measurement", issued to Kubo. The 

at the cellular provider, as described above with respect to preferred system and method for focus assessment is 

FIG. 3. Alternatively, other display devices, such as liquid described below 

crystal displays used for other purposes within the A focus score is computed for each video frame (i.e., each 

telephone, could be used as indicators. 45 captured image) Jf ^ focus &CQ[e exceeds a prede \ ermined 

The imager 100 also preferably has an on/off switch (not value, then it is determined that the image is focused enough 

shown), such as a pushbutton, for powering up the imager for further processing and comparison. If the focus score 

and initiating the image acquisition process. Power for the does not exceed the predetermined value, then it is deter- 

imager 100 is preferably supplied by a battery. The imager mined that the image is not focused enough for further 

100 receives and acts on instructions from the processor 210 50 processing, and an indicator (such as indicator 310, 

to perform functions such as lighting or turning off the described with respect to FIG. 5) is activated and a further 

indicator lamp(s) 305, providing the audible signals via the image is captured. Alternatively, a sequence of image frames 

speaker 310, and lighting the 'accept' and 'reject' indicators. can be obtained that cycle through a range of focus distances 

FIG. 6 is a more detailed flow chart of a method of strobed at the video frame-rate, and the focus score corn- 
operation in accordance with the present invention. The eye 55 puted for each frame can enable the selection of the best 
is illuminated at step 350 and an image of the iris is obtained focused frame within the sequence of frames. For example, 
at step 355 . At step 360, it is determined "ri tne image is by obtaining image frames at each of several different lens 
suitable for use with the image processing and comparison settings and then fitting a spline curve to their respective 
routines. If the image is suitable, the image is passed to the focus scores one can predict the lens position that would 
processor for further processing, at step 370, a nd transmis- 60 deliver substantially the sharpest focus, by setting the 
sion to the cellular provide r. A comparison of file template ' derivative of the parameterized spline curve to zero and then 
t&Tfi etemplates stored in a database atjhe cellular proN^idex. solving the equation for position, 

isnJen'ormed at step 371}. It' the comparison provides a Specific implementation features of the preferred focus 

positive match, then authorization is granted at step 376 for assessment system and method which enable its real-time 

the user to use the phone. If the comparison does not provide 65 operation, include (1) the computation of quantities in the 

a positive match, th en aumonzaiion is not_g mflifuLr:Qr 4he r 2D Fourier domain, without needing to compute an actual 

DseTto use the phone. 2D Fourier Transform of an image (this avoids the need for 
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approximately 2.25 million floating-point operations 
required for an FFT (Fast Fourier Transform) on a 500x500 
pixel image, as the computational complexity of an FFT on 
nxn data is 0(n 2 log 2 n)); (2) only 6,400 integer multiplica- 
tions (squarings) are performed, which in turn can be 5 
eliminated altogether by using small look-up tables; (3) no 
floating-point operations are required; (4) computation of 
focus scores is based upon simple algebraic combinations of 
pixel values within local closed neighborhoods, repeated 
across regions of the image; and (5) these operations not 
only allow the algorithm to execute in real-time, but it also 
enables a straightforward implementation in simple, low- 
cost, hardware devices that could be embedded within a 
digital camera or frame grabber. 

Preferably, the focus assessment processor 420 is fast 
enough to determine a focus score for each frame in a video 15 
image stream in less than the time it takes to acquire a new 
frame (e.g., approximately 25 ms). The frame-by-frame 
focus scores can be used to control a moving lens element 
for rapid and accurate focus control, or alternatively, to 
select which of several frames in a video stream is the one 20 
in best focus. The rapid selection of well-focused video 
frames for further processing, such as image analysis and 
pattern recognition, is important in real-time computer 
vision because it prevents wasting processing time on 
poorly-focused images. 25 

The preferred focus assessment processor measures the 
focus quality of video images at standard rates of 25 (PAL) 
or 30 (NTSC) frames per second. 

It is contemplated that the focus assessment processor 420 
can be implemented in a general purpose personal computer 30 
(PQ or by a dedicated, low cost processor which is small 
enough to be incorporated into the camera electronics. 

The processing of a video frame results in the return of an 
integer value (on a scale between 0 and 100) reflecting the 
quality of focus; the larger the value of the integer, the better 35 
the focus. A value of 0 indicates a completely defocused 
image whereas the value of 100 indicates maximum focus 
quality. A predetermined threshold is used to determine 
whether an image is sufficiently focused or whether another 
image needs to be retrieved. For example, values greater 40 
than about 40 can indicate sufficient quality of focus to 
warrant further image processing, while values less than 
about 40 cause a new image frame to be grabbed, and 
optional feedback provided to the focusing mechanism, if 
one exists, or to the subject controlling the camera position 45 
(via the indicator 310, for example). 

Optical defocus is a phenomenon of the 2D Fourier 
domain. An image represented as a 2D function of the real 
plane, I(x,y), has a 2D Fourier Transform F(o,v) defamed as 
shown in equation 1. 50 

55 

In the image domain, defocus is preferably represented as 
convolution by the 2D point-spread function of the defo- 
cused optics. This in turn may be modeled as a Gaussian 
whose space constant is proportional to the degree of 
defocus. Thus, for perfectly focused optics, the optical 60 
point-spread function shrinks almost to a delta function, and 
convolution with a delta function causes no change to the 
image. Progressively defocused optics equates to convolv- 
ing with a wider and wider point-spread function, which 
averages together whole neighborhoods of pixels by such a 65 
weighting function, thereby producing an increasingly 
blurred image. 



,699 Bl 

10 

If the convolving optical point-spread function causing 
defocus is modeled as a Gaussian whose width represents 
the degree of defocus, then defocus is equivalent to multi- 
plying the 2D Fourier Transform of a perfectly focused 
image with the 2D Fourier Transform of the "defocusing" 
(convolving) Gaussian. This latter quantity is itself just 
another 2D Gaussian but in the Fourier domain, and its space 
constant (a) there is the reciprocal of that of the image- 
domain convolving Gaussian that represented the optical 
point-spread function. The preferred focus assessment pro- 
cessor uses (1) the duality of convolution and multiplication 
in the two domains; (2) the fact that a Gaussian has a Fourier 
Transform which is itself a Gaussian, but with the reciprocal 
width because of (3) the Similarity Theorem. Thus, the 2D 
Fourier Transform \>J^i y \) of an image defocused to degree 
a is related to F(w,v), the 2D Fourier Transform of the 
corresponding in-focus image, as given by equation 2. 

From the above equation, the effect of defocus is to attenuate 
primarily the highest frequencies in the image, and that 
lower frequency components are virtually unaffected by 
defocus since the exponential term approaches unity as the 
frequencies (/z,v) become small. For simplicity, the present 
description has assumed isotropic optics and isotropic blur, 
and the optical point-spread function has been described as 
a Gaussian. However, the analysis can readily be generalized 
to non-Gaussian and to anisotropic optical point-spread 
functions. 

Thus, an effective way to estimate the quality of focus of 
an image is to measure its total amount of energy in the 2D 
Fourier domain at high spatial frequencies, since these are 
the most attenuated by defocus. One may also perform a 
kind of "contrast normalization" to make such a spectrally- 
based focus measure independent of image content, by 
comparing the ratio of energy in the highest frequency bands 
to that in slightly lower frequency bands. Such spectrally- 
based energy measurements are facilitated by exploiting 
Lord Rayleigh's theorem for conserved total power in the 
two domains, shown in equation 3. 

Thus, high-pass filtering or band-pass filtering an image at 
a ring of high spatial frequency (using only convolution in 
the 2D image domain) and measuring the residual energy, is 
equivalent to making the corresponding energy measure- 
ment in the high frequency bands of the 2D Fourier domain. 
The appropriate measurements in the 2D Fourier domain to 
assess focus can be performed without computing a time- 
consuming 2D Fourier Transform. Indeed, the measure- 
ments can be performed without even a single floating-point 
operation, and even without any multiplications if appropri- 
ate convolution kernels and look-up tables are used. 

A real-time procedure for focus assessment based on these 
theoretical principles is used in the focus assessment pro- 
cessor 420. It executes much faster than the video frame- 
rate, and so real-time focus assessments can be made on a 
frame-by-frame basis. These can be used either to control 
the position of a focusing lens element, or alternatively as a 
type of autofocus system in which frames are grabbed at a 
variety of focal depths in order to select only the best one for 
processing, or to prevent time being wasted on processing 
image frames which are assessed to be in poor focus. 
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The 2D spectral measurements described above can be 
implemented by convolving an image with the following 
convolution kernel, in which pixel values within a prede- 
termined region, such as, for example, an (8x8) 
neighborhood, are added together with the weights indicated 5 
in each of the cells: 
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It should be noted that no pixel-by-pixel multiplications are 
needed in order to impose these weights. Rather, the pixels 
in the central region are added together, such as the (4x4) 
square, that sum is tripled, and then all pixel values in the 20 
outer two pairs of rows and columns are subtracted from the 
tripled sum. The result is squared and added to an 
accumulator, thus implementing the left-hand side of equa- 
tion (3) above for this local region of the image. The 
complete (8x8) convolution kernel is then moved to a new 25 
position in the image, along a sampling grid that selects 
every 4th row and every 4th column, and the operation is 
repeated. Thus, to assess the quality of focus within the 
central (320x320) region of an image, this set of 64 pixel 
summations followed by a squaring operation is repeated a 3Q 
total of (320/4) 2 =6,400 times. 

In the 2D Fourier domain, the spectral consequences of 
this operation can be appreciated by examining the 2D 
Fourier Transform of the convolution kernel above. The 
kernel is equivalent to the superposition of two centered 
square box functions, one of size (8x8) and amplitude -1, 35 
and the other of size (4x4) and amplitude +4 (for the central 
region in which they overlap, the two therefore sum to +3). 
The 2D Fourier Transform of each of these square functions 
is a 2D "sine" function, whose size parameters differ by a 
factor of two in each of the dimensions and whose ampli- 40 
tudes are equal but opposite, because the two component 
boxes have equal but opposite volumes. Thus, the overall 
kernel has a 2D Fourier Transform K(/i,v) which is the 
difference of two differently-sized 2D sine functions, as 
given by equation 4. 45 

sin(/i)sin(v) sm{2/i)sin(2v) (4) 
K<jj< v) = > 

50 

This is a high-pass (or ultimately a band-pass) filter, 
selecting only a high range of spatial frequencies in all 
orientations. Towards its center, corresponding to very low 
spatial frequencies, its value approaches zero (as can also be 
inferred from the fact that the sum of all pixel weights in the 55 
convolution kernel shown above is zero). Thus, low fre- 
quencies play little or no role in computing a focus score, 
and only relatively high frequencies contribute significantly 
to the computation of a focus score. Equation (3) shows that 
summing the squares of all the local convolution sums eo 
across the image is equivalent to summing the total amount 
of high frequency energy in the 2D Fourier Transform of the 
image. The action of the convolution kernel is to impose the 
above power spectral weighting function so that primarily 
high frequency energy is measured. 65 

Finally, the summated 2D spectral energy is passed 
through a compressive nonlinearity of the form f(x)=100 
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xV^+c 2 ) in order to generate a normalized focus score in 
the range of 0 to 100 for any image. 

The focus assessment technique is applied immediately 
after each image frame is digitized and stored in the frame 
buffer memory 410 in order to assess whether the focus 
quality is sufficient to warrant any further processing. If the 
calculated focus quality value of the captured image is 
greater than or equal to a predetermined value, the image is 
passed to applicable programs for further processing, for 
example for extraction of a biometric template. The focus 
assessment technique can be used to compare the relative 
focus of an entire series of images in order to select the one 
most in-focus (i.e. having the highest focus assessment 
score), as well as to measure a single image. 

The focus assessment technique can be used to provide a 
feedback indication to a system user who controls the 
position of the imager relative to the object being imaged. 
This can be accomplished by activating an indicator which 
would continue, while successive images are captured and 
their focus assessed, until the focus assessment score 
exceeds a predetermined value. At this point, the indicator is 
deactivated and the last image captured is transferred to the 
image processor 210 where it is processed to extract the 
biometric template. 

The application of the focus assessment technique in 
combination with the feedback indicator helps resolve the 
man-machine interface problems associated with the use of 
digital imaging devices on the eye. Individuals using the 
system are provided positive, objective indicators and feed- 
back as to the quality of image focus. The focus assessment 
processor can also be used in any situation where it is 
required to determine the quality of focus of video images at 
industry standard frame rates (NTSC and PAL), 

Thus, the image is obtained at the imager and transmitted 
to an analog to digital converter 405. The digitized video 
information is then stored in a frame buffer memory 410, 
The focus assessment processor 420 isolates the central 
320x320 region of the image. 8x8 pixel blocks (each pixel 
is in only one block) are then processed by first summing 
pixels in the central 4x4 region, tripling that sum, and then 
subtracting from this value all the pixel values in the outer 
two pairs of rows and columns. This result is then squared. 
This process is performed on each 8x8 block, and the results 
are summed. After the entire image has been processed, the 
summed result is compressed nonlinearly to generate a focus 
score between 0 and 100. This score is then compared to a 
predetermined number for determining if the indicator 310 
should be activated. 

The focus assessment is preferably performed by the 
microprocessor 210, or it can be a separate processor ele- 
ment within the telephone. 

It is contemplated that in addition to the focus assessment 
processor, an auto-focus lens system could be used in the 
present invention. The results of the focus assessment con- 
trol the lens system, thereby automatically adjusting focus to 
produce an optimal image. This would place less of a 
premium on the accuracy with which the user positions the 
eye, and would be helpful if the user could not see or hear 
the indicators described above. 

The iris imager of the present invention can be used as a 
security module for electronic devices such as a telephone. 
FIG. 8 is a schematic diagram of an exemplary iris imager 
incorporated into a telephone in accordance with the present 
invention. The imager 700 comprises the camera 105, lens 
110, mirror 120, and illuminator 130, as described above 
with respect to FIG. 1. The imager 700 also comprises 
visible indicators 555, 556, 557, which are similar to indi- 
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cators 305, 306, 307, respectively, described above with 
respect to FIG. 5. An audible indicator 560, similar to 
indicator 310, is also disposed within the imager 700. The 
imager 700 further comprises electronics and circuitry 500 
for processing and comparing the obtained image. The 
electronics and circuitry 500 comprises a microprocessor 
510 (similar to microprocessor 210) that controls the imager 
700 along with an I/O controller 525 and runs software held 
in a ROM 505. The processor 510 is connected to the ROM 



505, a RAM 532 that is capable of storing at least oDe 10 claims. 



can be returned to its locked, secure state either upon being 
powered down or upon completion of the phone call. 

Although illustrated and described herein with reference 
to certain specific embodiments, it will be understood by 
those skilled in the art that the invention is not limited to the 
embodiments specifically disclosed herein. Those skilled in 
the art also will appreciate that many other variations of the 
specific embodiments described herein are intended to be 
within the scope of the invention as defined by the following 



captured image or an iris, another memory 530, such as an 
EPROM, for storing a plurality of biometric templates or iris 
images that are to be compared with the captured iris image. 
The electronics and circuitry 500 is also connected to the 
camera 105, the illuminator 130, and the indicators 555, 556, 15 
557, 560 for controlling these elements of the imager 700. 
The processor can also comprise a focus assessment 
processor, similar to the focus assessment processor 420. 

It should be noted that in the embodiment of FIG. 8, the 
database memory 530 of templates is stored within the 20 
imager 700 and not at a central station (as described, for 
example, with respect to FIG. 4), as is the processor 510 
used in the comparison. In the embodiment of FIG. 8, the 
comparison of the captured image template with the stored 
templates takes place locally within the telephone, and no 25 
data is sent to the central station for comparison or authen- 
tication. 

The imager 700 is coupled to telephone electronics 570 
for transmitting encrypted or unencrypted data to another 
telephone or system via an antenna. The telephone electron- 30 
ics 570 is essentially a telephone and is preferably a con- 
ventional cell phone having telephone electronics and is 
connected to a transmission antenna. Preferably, a conven- 
tional voltage regulator (not shown) provides the appropriate 
operating voltage to the imager 700 from the power supply 35 
(e.g., a battery) of the phone. 

Preferably, the imager 700 of the present invention is 
incorporated into a handset of a telephone 575, as shown in 
FIG. 9. The present invention can be incorporated into a 
conventional digital cell phone, as shown in FIG. 9, such as 40 
those manufactured by Qualcomm. 

FIG. 10 is a flow diagram of an exemplary method of 
operation of a telecommunications iris imager in accordance 
with the present invention. A user desiring to make a 
telephone call first unlocks the telephone by having his iris 45 
identified by the imager residing within the phone. The eye, 
and thus the iris, is illuminated at step 605. An image is 
obtained of the iris at step 610. At step 615, it is determined 
if the image is suitable for further processing and 
comparison, as described above. If the image is not suitable, 50 
the appropriate indicators are activated at step 620, and 
processing returns to step 610 with the capture of another iris 
image. 

If the captured image is suitable for further processing, the 
image is processed at step 630 (an indicator can be activated 55 
to alert the user that a suitable image has been captured) and 
is compared to the stored images residing in a database, for 
example, in a memory, at step 635. If there is no match 
between the captured image and the stored images, at step 
640, the appropriate indicators are activated at step 645, and 60 
the imaging routine exits. In this manner, the telephone 
remains locked, and cannot be used because it is determined 
that the user is unauthorized. 

If there is a match between the captured image and the 
stored images, at step 640, the phone is unlocked (and an 65 
indicator can be activated) at step 650, and the user is then 
able to use the phone to place a call, for example. The phone 



What is claimed is: 

1. A telephone security module comprising: 

iris acquisition means having a front surface for obtaining 

an image of an iris of an eye; 
a lens having a image plane disposed in front of said front 

surface of said iris acquisition means; 
a mirror disposed on a side of said lens opposite said iris 

acquisition means; 
an illuminator disposed along a side of said mirror; 
a memory for storing an iris image obtained by said iris 

acquisition means; 
a processor for extracting a template from said stored iris 

image; and 

a communications interface for transmitting said template 
to a central station. 

2. The telephone security module according to claim 1, 
wherein said iris acquisition means comprises a camera, and 
said mirror is a cold mirror. 

3. The telephone security module according to claim 2, 
wherein said camera is sensitive to light having a wave- 
length in a range between about 400 nm and about 1100 nm. 

4. The telephone security module according to claim 2, 
wherein said mirror reflects light having a wavelength in a 
range between about 400 nm and about 700 nm and passes 
light having a wavelength greater than about 700 nm. 

5. The telephone security module according to claim 1, 
wherein said illuminator emits light having a wavelength in 
a range between about 680 nm and about 900 nm towards 
the iris of the eye being imaged, the eye being out of contact 
with the iris imaging apparatus. 

6. The telephone security module according to claim 1, 
further comprising at least one of a visible indicator and an 
audible indicator to indicate when the image of the iris has 
been obtained. 

7. The telephone security module according to claim 6, 
further comprising a focus assessment processor coupled to 
said at least one of a visible indicator and an audible 
indicator. 

8. The telephone security module according to claim 1, 
wherein said processor unlocks a telephone responsive a 
signal received from said central station. 

9. A system of identification of a person for unlocking a 
telephone, comprising: 

the telephone security module of claim 1; 

a second memory for storing at least one template of at 

least one image of an iris of at least one person's eye; 

and 

a second processor for comparing said template of said 
stored iris image with said at least one stored template 
of said second memory to identify the person, and for 
unlocking the telephone responsive to the result of said 
comparison. 

10. The system according to claim 9, wherein said second 
memory and said second processor are housed in a central 
station remote from the telephone security module. 
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11. The system according to claim 9, wherein said second 22. The telephone security module according to claim 20, 
processor unlocks the telephone when said template of said wherein said mirror reflects light having a wavelength in a 
stored iris image substantially matches said at least one range between about 400 nm and about 700 nm and passes 
stored template of said second memory. ljght haviag & wavclcngth grcatcr man about 70 0 nm. 

12. A telephone security module comprising: 5 2y ^ ^hont security module according to claim 20, 
ins acquisition means having a front surface for obtaining wherein said illuminator emits light having a wavelength in 

an image of an iris of an eye; a raQge bctwecn about 680 nm and about 900 nm towards 

a lens having a image plane disposed in front of said front the iris of me bein i ^ the bein Qut of COQtact 

surface of said iris acquisition means; ^ ^ ^ ^ imaging 

a mirror disposed on a side of said lens opposite said ins 2 4. The telephone security module according to claim 20, 

acquisition means; further comprising at least one of a visible indicator and an 

an illuminator disposed along a side of said mirror; audible indicator to indicate when the image of the iris has 

a first memory for storing at least one template of at least been obtained. 

one image of an iris of at least one person's eye; is 25. The telephone security module according to claim 24, 

a second memory for storing an iris image obtained by further comprising a focus assessment processor coupled to 

said iris acquisition means; said at least one of a visible indicator and an audible 

a processor for extracting a template from said stored iris indicator. 

image; and 26. The telephone security module according to claim 20, 

a comparator for comparing said template from stored iris 20 w h ere in sa i d processor unlocks a telephone responsive a 

image with said at least one template to identify the signal received from said central station. 

person. 27. A telephone security module comprising: 

13. The telephone security module according to claim 12, a camera having a front surface for obtaining an image of 
wherein said iris acquisition means comprises a camera, and ^ an iris of an eye; 

said mirror is a cold mirror. a ler^ having a image plane disposed in front of said front 

14. The telephone security module according to claim 13, of ^ cam 
wherein said camera is sensitive to light having a wave- 
length in a range between about 400 nm and about 1100 nm. a 001(5 mirror disposed on a side of said lens 

15. The telephone security module according to claim 13, 3Q opposite said camera; 

wherein said mirror reflects light having a wavelength in a an illuminator disposed along a side of said minor; 

range between about 400 nm and about 700 nm and passes a nrst memory for storing at least one template of at least 

light having a wavelength greater than about 700 nm. one image of an iris of at least one person > s eye; 

16. The telephone security module according to claim 12, , * . • • • . .... 
wherein said illuminator emits light having a wavelength in 3S a SeC * nd mcmor y for storin S aD ms ima * c obtained ^ 
a range between about 680 nm and about 900 nm towards said camera i 

the iris of the eye being imaged, the eye being out of contact a processor for extracting a template from said stored iris 

with the iris imaging apparatus. image; and 

17. The telephone security module according to claim 12, a comparator for comparing said template from stored iris 
further comprising at least one of a visible indicator and an ^ image with said at least one template to identify the 
audible indicator to indicate when the image of the iris has person. 

been obtained. 28. The telephone security module according to claim 27, 

18. The telephone security module according to claim 17, wherem said camera is tQ u ht hayin a waye _ 
turner comprising a focus assessment processor coupled to { ^ in a betweefl abom ^ nm afld abmU n0Q nm 
said at least one or a visible indicator and an audible A . -, Q „ . . . .. , . t , . 
indicator telephone secunty module according to claim 27, 

m 19/£Se telephone security module according to claim 12, where !° f aid ^T^f 5 Ugh ' ^ a wavelength in a 

wherein said processor unlocks a telephone responsive to the [ an l ge between abou 400 L Qm and about ™ nm and P asses 

results of said comparison. h S ht havin S a wavc * c ngto greater than about 700 nm. 

20. A telephone security module comprising: „ 30 - J** telephone security module according to claim 27, 

c * -r r u* ■ • * r wherein said illuminator emits light having a wavelength in 

a camera having a front surf ace for obtaining an image of , , , i rar , * * * rt B , 

Qn • • r & a range between about 680 nm and about 900 nm towards 

dii ii io oi au eye, , . ... 

, . . . i j. . ■ r c *j c the ins of the eye being imaged, the eye being out of contact 

a lens having a image plane disposed in front of said front wim ^ { ^ { ^ apparatus, 

surface of said camera; n * i u •* ^ i . * • ~„ 

, , . , , , „ . , , ~ 31 ■ The telephone security module according to claim 27, 

a concave cold mirror disposed on a side of said lens 55 at ^ Qne Qf a and aQ 

opposite said camera; audibk indicator t0 mdicatc whcn the imagc of mc iris has 

an illuminator disposed along a side of said mirror; becn obtained. 

a memory for storing an iris image obtained by said 32. The telephone security module according to claim 31, 

camera; 60 further comprising a focus assessment processor coupled to 

a processor for extracting a template from said stored iris said at least one of a visible indicator and an audible 

image; and indicator. 

a communications interface for transmitting said template 33. The telephone security module according to claim 27, 

to a central station. wherein said processor unlocks a telephone responsive to the 

21. The telephone security module according to claim 20, 65 results of said comparison, 
wherein said camera is sensitive to light having a wave- 
length in a range between about 400 nm and about 1100 nm. * * * * * 
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ABSTRACT 



A person identifie r and management system is described. 
The person identifier and management system includes a 
face image database that stores the face feature data of each 
of a number of face images and the person-identifying data 
associated with each of the face images. A camera^is-also 
included in the person identifier and management system for 
captunng an input face image. A face analysis system is then 
coupled to the camera and the image "database to extract the 
faceTeature data* of the input face imagelnffto compare the 
face feature data of the input face image with that of each of 
the face images stored in the database such that the person- 
identifying data of any stored face image similar to the input 
face image can be retrieved from the image database. A 
method of retrieving person-identifying information of an 
input face image from a face image database is also 
described. 

7 Claims, 7 Drawing Sheets 
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1 2 

PERSON IDENTIFIER AND MANAGEMENT prior art typically causes the computer to compare the image 

SYSTEM pixel data of the input face image with the corresponding 

pixel data of each of the images stored in the computer. The 

BACKGROUND OF THE INVENTION comparison typically takes a very long time to complete and 

. 5 uses a lot of system resources (e.g., memory and commu- 

1. Field of the Invention nication bandwidth). This is because the computer has to 

The present invention pertains to image processing and compare each image pixel data of the input face image with 

information management. More particularly, this invention the corresponding image pixel data of each of the stored face 

relates to a person identifier and management system that images. A match occurs when each pixel data of the input 

can retrieve person-identifying informa tion from a data base 10 face image is similar the corresponding pixel data of a stored 

based on an input face such that the input fa ce-can-be face image. Using this method, unless the two compared 

Jdentjfied. images are extremely similar, it is very difficult to find a 

~XT5escription of the Related Art match between any two face images. This is especially true 

As is known, human nature typically shows that people because noise and other positional errors may be introduced 

want to be recognized by others. Thus, it is important for 15 into the image data when the input face image is captured, 

people to recognize and recall the names of other people SUMMARY OF THE INVENTION 

they have previously met or acquainted during meetings. ~ f . , . 

• i a »• .u • * T . • • ST One feature or the present invention is to help a user recall 

social functions, or other occasions. Not recognizing the . , f. . , r 

person before you may affect you adversely. For example, if 01 f °°f 1ZC F pcoplc * C ^ has P™™™^ 
a superior or employer does not recognize or cannot recall 20 Another feature of the present invention is to allow 
the name of an inferior or employee, it may give the [ etneval of personal-identifymg information from a data- 
inferior/employee the impression that the supervisor/ basc m accordance ™ th an face ««ge- 
employer does not care about the inferior/employee, or the A f arther feature of tne P resent invention is to provide a 
inferior/employee is not important in the organization. In P erson "kn^fier and management system that searches for 
addition, if a business person cannot recognize or recall the 25 similar faces of an input face in a face image database such 
name of a business counterpart from another organization, it mat person-identif ying information associated with the simi- 
may affect the business dealing or potential business dealing Ur faces can be retrieved from the database to identify the 
with the organization that business counterpart represents. m P ut ^ ace * 

However, for people who have a relatively large number AstiU fea ^ re ° f ^ Pj esent *™*ian * to Provide 

of casual acquaintances, association of faces with names is 30 a P^f?™ 1 P erson t lden , ufier and management system that is 

a difficult memory game. In addition, some people are just P ortable > P ersonal > and eas V to use - 

not good at recognizing or recalling names of formal A person identifier and management system is described, 

acquaintances, especially when the formal acquaintances ^ P erson identifier and management system includes an 

have only been briefly introduced to you. Change of envi- ima S e data base that stores face feature data of each of a 

ronment may also cause problems for people to recognize or 35 aumber of face images and person-identifying data associ- 

recall names of others. For example, a person will have more ated wm each of {he face images. An image capturing 

difficulty in recognizing people initially if the person is system is also included for capturing an input face image. A 

surrounded by a large number of strangers (e.g., in a new face anal Y sis svs f em is then coupled to the image capturing 

office building). system and the image database to locate and retrieve the 

*u a e j • .40 person-identifying data of any stored face image similar to 

With the advance of computer and image processing f u . , c ■ * L - j * i- r 

t u i * •* ■ * * . the input face image from the image database. The face 

technologies, it is now possible to store an image in a , . , , c * V 

* \ tl- « • j . * j 1 . analysis system does this by. for example, extracting the face 

computer system. This can be done using a digital camera to e J a . r • r • *\ ' 6 , * 

capture an image. In addition, a scanner can be used to f * °! * e UpU f / Ce im *& an f U T?T & 2" 

convert a printed image into digital form. The digital image M ^ eatUre data °f *e input face miage w,th that of each of the 

i j • ■ i / j . \ j • • rrZ 45 face images stored in the database, 

includes image pixels (or dots) arranged in a matrix. The . 7 . f • 

data for each image pixel are then stored in the computer. r A me } hod of rct ? cvin g person-identifying information 

The digital image can be retrieved for display from the from a facc ima S c database bascd on an mpul face image 

computer includes the step of extracting face feature data from the 

rr* • ii c . . 4 . , t_ *i * i n P ut f acc image. The face feature data of the input face 

This allows a face image database to be built in a S0 image is then compared with that of a number of face images 

computer system that can help the user to manage the stored m ^ da , abase ^ of ^ stored ftce 

acquaintances intormaUon. Ihe tace image database typi- includes the person -identifying information that identifies 

tally • stores a number of face images and Uie person- me jve face ■ , f ^ fece d ^ rf one 

identifying informaUon associated with each of the stored the stored face im matches ^ of ^ ^ f&ce . 

images The person-identifying informaUon islypically a ss then person . identifying information of that stored flee 

text information that identifies the face. The person- • a t *u • j * i_ * -j »-r *i_ 

4 -r * * c *■ ■ * * c . • i j image is retrieved from the image database to identify the 

identifying information in text format may include name, - n JJ f ace j ma „ e 

contact information, date and occasion of last meeting or all V^, c 4 , , r i 

the prior meetings. When the user retrieves a face image ° th " fcatUrcS and ^ v «? t »8» f the present invention 

from the database, the person-identifying information asso- 60 will become apparent from the following detailed 

ciated with that image is also retrieved from the database. descnpUon taken in conjunction w,th the accompanying 

When the person-identifying informaUon is displayed with drawings illustrating by way of example the principles of 

the associated face image, the user can then recognize the C mVCn l0D * 

face and recall the name of the face, BRIEF DESCRIPTION OF THE DRAWINGS 

However, searches within this face image database is still 65 FIG. 1 shows the structure of a person identifier and 

nearly impossible. In order to identify a desired face image management system that implements one embodiment of the 

stored in the computer based on an input face image, the present invention. 
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FIG. 2 illustrates the perspective view of the person FIG. 1 shows in block diagram form the structure of the 
identifier and management system of FIG. 1 as a portable, person identifier and management system 10. The person 
personal, and hand-held system in accordance with one identifier and management system 10 can be a single stand- 
embodiment of the present invention. alone device or implemented in a computer or computer- 

FIG. 3Ais a front view of the personal person identifier 5 enabled system. If the person identifier and management 

and management system of FIG. 2. s y stem 10 15 implemented in a computer system, the com- 

t-f^ ™ L iL . • r .i i P uter system can be a personal computer, a notebook 

FIG. 3B shows the top view of the personal person a mini-computer, a hand-held computer, a per- 

identificr and management system of FIG. 2. sonal commurucator , a workstation, a network computer, a 

FIG. 4 shows the analysis system of the person identifier ^ ser ver computer, or a mainframe computer, 

and management system of FIG. 1. ^ can be secn from nG t the pcrson identifier and 

FIG. 5 shows the input/output system of the person manapemen tj>vstem 10 includes the input/output system 11 . 

identifier and management system of FIG. 1. the analysis system 12 . and the profile database 13 . The 

FIG. 6 shows the data format for each of the face images input/output system 11 is connected to the analysis system 

stored in the profile database of the person identifier and l5 12. The analysis system 12 is then connected to the profile 

management system of FIG. 1. database 13, The input/output system 11 is used provide the 

DETAILED DESCRIPTION OF THE ^ and ^ for 1 thc ^ ntificr j nd 



INVENTION 



management system 10. The analysis system 12 is used to " 7 -yydld^ 
detect a face from an mpufimage and to use the face feature f ^° f 
FIG. 1 shows a person identifie r and management system ^ data extracted from the input race i mage to search the profile, f Jkin^ 60 



10 that implements one embodiment of the present inven- database 13 in order to lclenTi fyThe input face image. 
tion. As will be described in more detail below, the person "-TTTg^ fiCTmar5a^ 3"i bres a number of face inug e<T"$ / JL> JLfle ^ 
identifier and management system 10 allows a user to_ and the iTface feature data and person-identifying or face- ' 
retrieve person-identifying or face-identifying information ^geWyiriFda ta. The face feature data and the person- 
ot a person from a profile database (o r face image database^ ^ identifying data are fully searchable in the database 13. The 
13 Of the ptfisuu Identifier and management system 10 based face feature data and its creation will be described in more 
on in e^pe rs on^kce inguj^d to th e person identifi er and detail below, also in conjunction with FIGS. 1 and 4. The 
maiiagBBeflfsystemlO such that the person can be ident i- person-identifying data can be a text data, an image data, and 
fled or recognizee by tie user of the person identifying a combination of text and image data. The person- 
system lu. this means that the person identifier and man- ide ntifying dat a o f a face image may include the name of the 
agemenf system 10 can use the input face image to sear ch person, amnation or employment ot the person, and time and 
t he profile database 13 f or matching face or faces and their occasion of last meeting or all of the previous meetings with 
associated taceH aej^ifi>ing^jperson-ia^ntifying data so that the person. The employment information may include name 
the input face c an De identified or recognized. 0 f the company the person is currently working for, the title 
As will be described in more detail below, the person 35 and position of the person within the company. In addition, 
identifier and management system 10 includes the profile the person-identifying data may also include other 
database 13. Hie profile database 13 stores a number of face information, such as telephone and fax numbers and/or 
images and their associated face-identifying or person- e-mail address, the name of the third person who introduced 
identifying data. The profile database 13 stores the pixel data the person, and/or other types of person-identifying infor- 
of each of the face images. In addition, the profile database 40 mation. 

13 also stores the face feature data of each of the face Each of the digital face images stored in the profile 

images. The face feature data of a face image is associated database 13 is, for example, in the pixel matrix format. This 

with that face image in the profile database 13. The person- means the profile database 13 stores the pixel data of each 

identifying data of each of the face images is also associated of the stored face images. In addition, the face feature data 

with the respective face image in the profile database 13. 45 and the person-identifying data of each face image are 

In addition to the profile database 13, the person identifier associated with or linked to the respective face image stored 

and management syste m lO^also includes an analysis system in the profile database 13. The face feature data and the 

12 that (1) d etects if a n input imag e of the person identifier person-identifying data of a face image are, however, not 

STTa management system 10 contains a face image a nd (2) necessarily physically stored adjacent to the respective face 

extracts the face feature d ata from the input image if the 50 image in the profile database 13. The profile database 13 can 

input image is detected to contain a face . The input image is be implemented and accessed using known database tech- 

supplied by an input/output system 11 connected to the nologies. FIG. 6 shows the logical memory map 100 for the 

analysis system 1 2. Th e input/output system 11 is also use d face images stored in the profile database 13. 

tor outputting the person-identity ing data retrieved from the. As can be seen from FIG. 6, the logical memory map 100 

profile database_,U : — " 55 0 f the profile database 13 of FIG. 1 includes a number of 

The analysis ^s ystem 12 a ccesses the profile database 13 entries 101 through 100«, each for storing one digital face 

with thTextractedface feature data of the input face imag e image. Each entry (e.g., the entry 101) includes a face 

to locate the matching face feature data in the profile feature data field (e.g., the face feature data field 101a), a 

&UahaseJ3. When there is a match between the face feature person-identifying data field (e.g., the person-identifying 

"data of the input face image and the face feature data of one 60 data field 1016), and the image pixel data field (e.g., the 

or more stored face images, the person-identifying data of pixel data field 101c). The pixel data field of each entry 

the matching face images are retrieved from the profile stores the pixel data (i.e., the image data) of a face image 

database 13. This allows the user of the person identifier and stored in the profile database 13. The face feature data field 

management system 10 to identify or recognize the input of each entry stores the face feature data of a face image 

face image. The person identifier and management system 65 stored in the database 13 and the person- identifying data 

10 will be described in more detail below, also in conjunc- field of each entry stores the person- identify ing data of a 

tion with FIGS. 1-6. face image stored into the database 13. The face feature data 
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field of an entry is associated with or linked to the face image If no matching face feature data is found in the profile 

of that entry. The person identifying data field of an entry is database 13, it means that none of the stored face images in 

also associated with or linked to the face feature data of that the profile database 13 is similar to the input face image. In 

entry. this case, the analysis system 12 makes such indication to 

Referring back to FIG. 1, the input/output system 11 is 5 tnc us"- Tnc can then havc toe option of storing the 

used to input face images to the person identifier and m P ut facc image in the profile database 13, along with the^ 

management system 10. In addition, the input/output system associated face feature data and the person- identifying data . 

11 is also the output means of the person identifier and Be tore storing the input face image into the pro tile database 

management system 10 for data output. For example, the 13> the analysis system 12 asks the user to input the_ 

in put/output system 11 outp uts the person-identifying data 10 ^person-identifying data of the input face image via , the 

of a matching face stored in the person identifier and i nput/output system 11 . The structure and tunction of the 

management system 10 by displaying or printing the person - analysis system 11 will be described in more detail below, 

identifying dat a. The inp"ffl/output system 11 may include a also in conjunction with FIG. 4. 

number ol input and output devices or modules. For Referring to FIG. 4, the analysis system 12 includes a user 

example, the input/ output system 11 may include a display 15 interface 41, a search engine 42, and a face detection and 

t hat displays the input lace imag e to be searched in the feature extraction system 43. The user interface 41 is con- 

prorile database 13 for matching faces. The display also nected to the search engine 42 and the face detection and 

displays any person-identifying data retrieved from the feature extraction system 43. The search engine 42 is also 

profile database 13. In addition, the input/o utput system 11 connected to the face detection and feature extraction system 

may also include a digital camera th a t captures tne input face 20 43. Each of the components 41-43 is a functional module 

image. P lne digital camera may be replaced, for example, and can be implemented by software, hardware, and firm- 

wun a scanner. Moreover, the input/output system 11 may ware. 

include a text input device (e.g., keyboard or touch sensitive nc user interface 41 is an interactive user interface that 

s creen with han flwrlungTecognition capabilities^ that allows ^ part of m interactive window interface. The user interface 

i nput or lexi person-iden titying data into the people database 25 41 inchldcs a window ( not snown ) that displays the input 

U. tiu. 5 snows one implementation of the input/output facc image md the retr i e ved person-identifying data. In 

system 11 of FIG. 1, which will be described in more detail addition, the user interface 41 also includes a number of 

De J° w - command or function buttons that can be activated using a 

As can be seen from FIG. 5, the input/output system 11 user input device (e.g., mouse or sensitive pen). The user 

includes an I/O controller 51 connected to a number of I/O 30 interface 41 may also include a touch sensitive screen 

devices 52 through 56. The function of the I/O controller 51 keyboard or a hand writing recognition system. The user 

is to control various input/output activities of the input/ interface 41 can be implemented using any known technol- 

output system 11. The I/O controller 51 is implemented ogy, 

using known VO controller technologies. ^ ^ face detection and feature extraction system 43 

The I/O devices 52-56 include a camera 52, a display 53, receives the input face image from the user interface 41. The 

a scanner 54, a sensitive pen 55, and other I/O devices 56. face detection and feature extraction system 43 detects if the 

The camera 52 can be any known digital camera. The input face image contains any face. The detection system 43 

display 53 can be any known computer or TV display. In one also detects the eye locations of the face in the input face 

embodiment, the display 53 is a touch sensitive screen ^ image. The system 43 then uses the eye location data to 

display that allows text and/or graphics input using a sen- compute the face feature data of the detected face. The face 

sitive pen. In this case, the touch sensitive screen display feature data are also referred to as face coefficient data, 

may have an embedded keyboard or may have a handwriting Jn one embodiment, the detection system 43 uses neural 

recognition system. All of the I/O devices 52-55 can be network-based face detection technology to detect the face 

implemented using known technologies. 45 m thc input facc ^ ncural network-based face 

The other I/O devices 56 may include a keyboard , a detection scheme is disclosed in a publication entitled 

printer, and a modem or network ada ptor for network HUMAN FACE DETECTION IN VISUAL SCENES, by H. 

communications. A ll the other I/O devices 56 are known I/O Rowley (har@cs.cmu.edu), S. Baluja (baluja@cs.cmu.edu), 

devices. Although FIG. 5 show I/O devices 52-56 for the and T. Kanade (tk@cs.cmu.edu) in November 1995. The 

input/output system 11, the input/output system 11 may 50 publication is available from Carnegie Mellon University's 

include more or fewer than I/O devices 52-56. For example, Internet site at "www.ius.cs.cmu.edu/IUS/har2/har/www/ 

the input/output system 11 may not include the scanner 54, CMU-CS-95-158R/." Alternatively, other known face detec- 

Referring back to FIG. 1, the analysis system 12 of the tion technologies may be used. For example, a face detection 

person identifier and management system 10 is used to scheme described in U.S. Pat. No. 5,164,992, dated Nov. 17, 

detect the face from the input face image supplied from the ss 1992, and entitled FACE RECOGNITION SYSTEM can be 

input/output system 11. In addition, the analysis system 12 employed. 

extracts face feature data from the detected face in the input When the detection system 43 uses the neural network- 
face image. The analysis system 12 then uses the extracted based face detection scheme, the detection system 43 detects 
face feature data to search the profile database 13 for if the input face image contains any face by first applying a 
matching face feature data stored. If matching face feature so set of neural network-based filters to the image. The filters 
data is located in the profile database 13, then the analysis examine each location in the image at several scales, looking 
system 12 retrieves the person-identifying data associated for locations that might contain a face (i.e., looking for eye 
with the matching face feature data stored in the profile locations). The detection system 43 then uses an arbitrator to 
database 13. When the user of the person identifier and combine the filter outputs. The arbitrator is used to merge 
management system 10 views the retrieved person- 65 detections from individual filters and eliminate overlapping 
identifying data, the user is able to recognize or identify the detections. Using the neural network-based face detection 
input face image. technology makes the face detection robust, relatively fast, 
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and detect most faces. In addition, it allows the detection 
system 43 to detect different kinds of faces with different 
poses and lightings. 

Once a face is detected, the face detection and feature data 
extraction system 43 computes the face feature data of the 
detected face. In one embodiment, the face detection and 
feature extraction system 43 computes the face feature data 
of the detected face using a face feature extraction technol- 
ogy described in the above-mentioned U.S. Pat. No. 5,164, 
992. Alternatively, other known face feature extraction 
schemes may be used. 

The face feature extraction process by the system 43 is 
done by first normalizing, warping (i.e., rotating and 
scaling), and masking the face to produce a reference face 
that has a set of predetermined eye locations. This means 
that through normalizing, warping, and masking of the 
detected face, the eyes of the detected face are mapped into 
the predetermined eye locations of the reference face. This 
is done by translating, scaling, and rotating the detected face 
and by adjusting the brightness and contrast of the detected 
face. In other words, this is to standardize or normalize the 
detected face. This is disclosed in the above-mentioned U.S. 
Pat. No. 5,164,992. 

Then the system 43 projects the standardized face into an 
Eigeo-face space to compute the face coefficients or face 
feature data of the detected face. This is also disclosed in the 
above-mentioned U.S. Pat. No. 5,164,992. 

After the face detection and feature extraction system 43 
extracts the face feature data, the extracted face feature data 
is then sent to the search engine 42. The search engine 42 
then accesses the profile database 13 (FIG. 1) with the 
extracted face feature data. This is to find and locate any face 
feature data stored in the profile database 13 that matches or 
is similar to the extracted face feature data. 

The search engine 42 uses the extracted face feature data 
to search for face images stored in the profile database 13 
having similar face feature data. The search engine 42 
compares the extracted face feature data with each stored 
face feature data to output similarity scores based on the 
predetermined similarity measure. In one embodiment, the 
similarity measure includes a set of thresholds and the face 
feature data includes a set of values. The comparison of the 
corresponding values of two face feature data yields differ- 
ences between the values of the two data. The differences are 
then compared against their corresponding thresholds. A 
similarity score is obtained based on how many of the 
differences are less or more than the thresholds. The search 
engine 42 can be any kind of known search engine. 

The search engine 42 then retrieves the person -identifying 
data associated with the matching face feature data (i.e., the 
similar faces) from the profile database 13. In one 
embodiment, the search engine 42 retrieves both the person - 
identifying data and the pixel data of the similar images for 
display after a search. In another embodiment, the search 
and engine 42 selectively retrieves the person-identifying 
data of the similar images in accordance with their similarity 
scores. 

When there is no matching face feature data in the profile 
database 13 and the input face image is to be stored in the 
profile database 13, the face detection and feature extraction 
system 43 extracts the face feature data from the input face 
image and associates the face feature data with the input face 
image before storing the image and the face feature data into 
the profile database 13. In addition, the face detection and 
feature extraction system 43 receives the person -identifying 
data of the input face image from the user interface 41 before 
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storing the pixel data and the face feature data of the input 
face image into the profile database 13. The person- 
identifying data of the input face image is also associated 
with the face feature data of the input face image in the 

5 profile database 13. 

Referring back to FIG. 1, the person identifier and man- 
agement system 10 can be a portable personal system that 
can be easily carried around by the user of the system 10. For 
example, the person identifier and management system 10 

10 can be a hand-held personal system. Alternatively, the 
person identifier and management' system 10 can also be a 
non-portable system. FIG. 2 is a perspective view of a 
hand-held portable personal person identifier and manage- 
ment system 20 that implements one embodiment of the 

15 person identifier and management system 10 of FIG. 1. 
FIGS. 3 A and 3B show the front and top views of the 
hand-held portable personal person identifier and manage- 
ment system 20. 
As can be seen from FIGS. 2 and 3A-3B, the hand-held 

20 portable personal p erson identifier and manageme n t system 
20 includes a touch sensitive scree n display 22 that allows 
display ot the input f ace image 24 and the retrieved person- 
ldentitying data 25 and a digital camera 2 3 that is an 
integrated part in the hand-held portable personal person 

25 identifier and management system 20. The digital camera 23 
is used Jio capture th e input face image 24 . In addition, the 
hand-held portable personal person identifier and manage- 
ment system 2 0 also include a writing tooh21 that allows 
input into the hand-held portable personal person identifier 

30 and management system 20 via the touch sensitive screen 
display 22. 

In the foregoing specification, the invention has been 
described with reference to specific embodiments thereof. It 
35 will, however, be evident to those skilled in the art that 
various modifications and changes may be made thereto 
without departing from the broader spirit and scope of the 
invention. The specification and drawings are, accordingly, 
to be regarded in an illustrative rather than a restrictive 
sense. 

40 

What is claimed is: 

1. A person identifier and management system, compris- 
ing: 

(A) an image database that stores face feature data of each 
45 of a plurality of face images and person-identifying 

data associated with each of the face images, wherein 
the person- identifying data include name, affiliation, 
and time and occasion of meetings; 

(B) an image capturing system that captures an input face 

so ima S e i 

(C) a face analysis system coupled to the image capturing 
system and the image database to locate and retrieve 
the person-identifying data of any stored face image 
similar to the input face image from the image data- 

55 base; 

(D) a display system that displays the input face image 
and the person-identifying data of the stored face image 
determined to be similar to the input face image by the 
face analysis system, wherein the person identifier and 

eo management system is a single stand-alone portable 
hand held device that fits into a user's single hand, 
wherein the image database, the image capturing 
system, the display system, and the face analysis sys- 
tem all physically reside in the single stand-alone 

65 portable hand held device. 

2. The person identifier and management system of claim 
1, wherein the face analysis system locates and retrieves the 
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person-identifying data from the image database by extract- 
ing the feature data of the input face image and comparing 
the face feature data of the input face image with that of each 
of the face images stored in the face image database. 

3. The person identifier and management system of claim 
1, wherein the face analysis system further comprises 

(I) a face detection and feature extraction system that 
detects the input face image and extracts the face 
feature data from the input face image; 

(II) a search engine that compares the face feature data of 
the input face image with that of the face images stored 
in the image database. 

4. The person identifier and management system of claim 
3, wherein the face analysis system farther comprises a user 
interface that allows the input face image and the person- 



10 



10 



identifying data to be displayed on a display and that allows 
a user to input and associate the person-identifying data with 
a face image displayed. 

5. The person identifier and management system of claim 
1, wherein the image capturing system is an integrated 
digital camera. 

6. The person identifier and management system of claim 
1, further comprising a display that displays the person - 
identifyiDg data along with the input face image. 

7. The person identifier and management system of claim 
6, wherein the display is a touch screen display that also 
allows input of the person-identifying data into the face 
image database. 
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